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OFFICE ACTION SUMMARY 

D Responsive to communication(s) filed on -* 

□ This action is FINAL. 

□ Since this application is in condition for allowance except for formal matters, prosecution as to the merits is closed in 
accordance with the practice under Ex parte Quayle, 1 935 D.C. 1 1 ; 453 O.G. 213. 

A shortened statutory period for response to this action is set to expire 3 month(s), or thirty days, 

whichever is longer, from the mailing date of this communication. Failure to respond within the penod for response will cause 
the application to become abandoned. (35 U.S.C. § 133). Extensions of time may be obtained under the provisions of 37 CFR 
1.136(a). 
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□ Claims „ are sub i 0Ct t0 restriction or election requirement. 

Application Papers 

□ See the attached Notice of Draftsperson's Patent Drawing Review, PTO-948. 

I # : ■ Q The drawing(sj filed on is/are objected to by the Examiner. 

; j,^ □ the proposed drawing correction, filed on „ is □ approved □ disapproved. 

1 The specification is objected to by the Examiner, 

The oath or declaration is objected to by the Examiner. 
; Priority under 35 U.S.C. § 119 

□ Acknowledgement is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d). 
\ • □ All □ Some* □ None of the CERTIFIED copies of the priority documents have been 
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□ received in Application No. (Series Code/Serial Number) . ■ 

□ received in this national stage application from the International Bureau (PCT Rule 17.2(a)). 
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r □ Acknowledgement is made of a claim for domestic priority under 35 U.S.C. § 1 19(e). 
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DETAILED ACTION 



Claim Rejections - 35 U.S.C § 102 



1 . The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(e) the invention was described in a patent granted on an application for patent by another filed in the United 
States before the invention thereof by the applicant for patent, or on an international application by another who 
has fulfilled the requirements of paragraphs (1), (2), and (4) of section 371© of this title before the invention 
thereof by the applicant for patent. 

The changes made to 35 U.S.C. 102(e) by the American Inventors Protection Act of 1999 
(AIPA) do not apply to the examination of this application as the application being examined was 
not (1) filed on or after November 29, 2000, or (2) voluntarily published under 35 U.S.C. 122(b). 
Therefore, this application is examined under 35 U.S.C. 102(e) prior to the amendment by the 
AIPA (pre-AIPA 35 U.S.C. 102(e)). 

2. Claims 1-11, 13, 15, 17, 19, 20-24, 28-54, 58 and 62 are rejected under 35 U.S.C. 102(e) 
as being anticipated by Ahmed. 

Considering claims 1, 40, 43, 44, 55 and 58 Ahmed discloses a system for automatically 
processing a plurality of streams (multi info streams are met by 'audio', 'video' and 'text' see col 
33 lines 49-52) to classify event cues within the received data stream, wherein the event cues are 
text data, caption data, audio data and video data (see col 15 lines 25-42, col 17 lines 26-40, col 
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23 lines 10-45 and col 24 lines 18-63). The event cues detected are compared or matched to 
existing cues in specific classes or categories to classify the incoming data into an appropriate 
category or class (see col 17 lines 25-41 and col 18 lines 25-40). By searching through the 
classified cues, Ahmed discloses comparing and matching the unclassified cues with the classified 
cues to correctly classify and place the story segments in the correct category or classification. 
Thus Ahmed discloses selecting at least one contiguous portion of the multimedia presentation as 
a story segment, by matching a model of expected event cues with observed event cues in at least 
one information stream and to classify the presentations to which specific multimedia presentation 
belongs. The claimed story segment is depicted by Ahmed in fig 2-b. In particular figure 2-b 
depicts classifying news into a plurality of categories (see items 215 and 214) wherein a story 
segments are provided for each category (see "Erin: A Tropical Storm, etc.). Further Ahmed 
discloses extracting text data from a text information stream which corresponds to the story 
segment (see col 23 lines 35-59 and col 17 lines 25-41), extracting named entities from the text 
data (see col 23 lines 55-59 and col 15 lines 25-41) and extracting story summary data using the 
named entities as a basis (see col 1 5 lines 25 - 40 and fig 2-b, the news and sports stories retrieved 
are classified under "news" or "sports"). Further, Ahmed discloses displaying "keyframes" or 
"thumbnail" images (see col 16 lines 64-67) which is inherently extracted from the "imagery 
system" since it is displayed (see fig 2-b item 213 "Headline News"). Ahmed clearly depicts 
displaying the keyframe "Headline News" with the story summary data (see fig 2-b). Ahmed also 
discloses selecting a keyframe enabling a user to display a representative segment. In particular, 
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by selecting keyframes "Headline News" or "Sports" in fig 2-b a user can access a representative 
story segment(s). 

Considering claims 2 and 30, Ahmed inherently discloses a "machine" (met by system 
controller 103, data storage device and primary and secondary acquisition device in fig 1) which 
compare detected cues including transitions in data (met by "we go now to" or "update from" in 
col 24 lines 37-41) to existing or expected developed from observed event cues occurring in a 
class of media presentations cues to classify the detected cues and representative story segments 
in the correct category or class (see col 17 lines 25-41 and col 18 lines 25-40). 

Considering claim 3, Ahmed discloses the cues can be audio clues by detecting pause 
recognition in audio (see col 24 lines 58-60). Ahmed discloses the claimed "plurality of time 
difference based cues" because a large time difference in words or sounds in the audio would 
indicated a pause in the audio. 

Considering claim 4, Ahmed discloses the claimed "wherein the expected event cues 
comprise a plurality of intrastream cues taken from a given information stream" (see col 23 lines 
55-60 and col 24 lines 39-40, the cues "we go now to" and "update from" are taken cues take 
within the middle of the information stream or "intrastream"). 

Considering claims 6-11, Ahmed discloses detecting specific cues used in classifying data 
including text information, closed caption words, closed caption punctuation cues, token phrases, 
text strings (col 17 lines 24-41, col 23 lines 18-22, col 23 lines 54-59, col 24 lines 37-41). 
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Considering claim 13, Ahmed discloses the claimed "named entity" and "text string" (see 
col 23 lines 56-59). 

Considering claim 15, Ahmed discloses the claimed "named entity" and "closed caption 
punctuation clue" (see col 23 lines 55-59, note: punctuation met by comma's). 

Considering claim 17, Ahmed discloses the claimed "introductory news broadcast terms" 
(see col 24 lines 37-41, met by "update from"). 

Considering claims 19 and 22, Ahmed discloses the claimed "anchor to reporter hand-off 
phases (see col 23 lines 39-41). 

Considering claims 20, 21 and 23, Ahmed discloses the claimed "station" and "reporter 
named entity" (see col 23 lines 25-60). 

Considering claim 24, Ahmed discloses the claimed "leaders to highlights of upcoming 
news stores" (see col 24 lines 37-41, met by "more on that" which is a leader to an upcoming 
story). 

Considering claims 28 and 29, Ahmed discloses the claimed "named entities" (see col 23 
lines 55-59, see "Jane Doe" for name or "Anytown, USA" for location). 

Considering claim 31, Ahmed discloses the claimed "wherein the model of expected even 
cues is developed by statistical analysis of observed event cues" (see col 25 lines 30-67, col 28 
lines 35-55, col 29 lines 45-65). 
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Considering claim 32, Ahmed discloses taking cues from a video stream which inherently 
comprises images and thus discloses the claimed "expected event cues are taken from an image 
information stream" (see col 25 lines 30-40). 

Considering claim 33, Ahmed discloses the claimed "reporter frame" (see col 23 lines 55- 
59 and col 24 lines 37-41). 

Considering claim 34, Ahmed discloses the claimed "wherein the event cues are taken 
from an audio information stream" (see col 24 lines 42-46). 

Considering claim 35, Ahmed discloses a cues from audio pause recognition which meets 
the claimed "silence detection" (see col 24 lines 58-60). 

Considering claim 36, Ahmed discloses the claimed "news stories" (see col 15 lines 25 - 

42). 

Considering claim 37, Ahmed discloses cues which start a story segment (see col 24 lines. 
39-41 see "we go now to"). 

Considering claims 38 and 47, Ahmed discloses a method of processing representation of 
multimedia presentation having multi info streams (multi info streams are met by 'audio', 'video' 
and 'text' see col 33 lines 49-52) comprising the steps of summarizing segments of text, audio and 
video see col 33 lines 33-61) and thus discloses the claimed "selecting at least one contiguous 
portion of the multimedia presentation as a story segment". Further, since text data is extracted 
from the received stream and is used to compile the summary, Ahmed discloses "extracting text 
information from a text information stream corresponding to the story segment as text data" and 
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"extracting story summary data from the text data (see col 33 line 49 - col 34 line 39). Ahmed 
further discloses linking together a stored representation of the text data, summary data and name 
entity data (see fig b, "Erin: A Tropical Storm" meets the claimed 'named entity data' and 'text' 
and 'summary' data are met by the data description of "Erin: A Tropical Storm"). Ahmed also 
discloses displaying multiple summary elements representative of the extracted story segments 
(see fig 2-b which displays multiple stories wherein each story has multiple elements including the 
title, description, time and location). 

Considering claim 39, Ahmed discloses storing the all the data including summary data, 
named entity data for story segments ans one or more files one a file server (file server met by 
"data storage device 104" in fig 1). Further, the server is connected to the Internet which meets 
the "computer network" limitation (see col 1 1 lines 40-65). Further, since a user can "browse" 
the data and categories, Ahmed discloses the claimed "allowing access to the stored 
representations of news story segments available to a browser program running on the client 
computer connected to the network' (see col 1 1 lines 31-65). 

Considering claim 41, Ahmed discloses detecting the number of words occurring in a 
segment as a means for selecting a segment as a category or class (see col 28 lines 36-55) and 
thus discloses the claimed "determining a frequency of occurrence for the named entities per 
sentence in a story segment and selecting a sentence with a greatest named entity frequency of 
occurrence as a topic sentence". 
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Considering claim 42, Ahmed discloses detecting which segment has a greater similarity to 
a primary segment (see col 29 lines 18-44) including detecting the frequency or number of words 
or "greatest named entity frequency of occurrence" in a segment (see col 28 lines 26-55) and thus 
discloses the claimed "if more than one sentence has a greatest named entity frequency of 
occurrence, selecting the sentence closest to the beginning of the story as the topic sentence" 
limitation. 

Considering claim 45, Ahmed discloses wherein the claimed "wherein the story segment 
contains a reporter segment and the key frame is selected from the middle of the reporter segment 
(see col 24 lines 37-41, the phrase "we go now to" or "more on that" is the middle of a reporter 
segment). 

Considering claim 46, Ahmed discloses the claimed "wherein the story segment contains 
an anchor booth segment and the keyframe is selected from the middle of the anchor booth 
segment" (see col 24 lines 37-41, phrases "we go now to", "update from" or "more on that" 
meets the claimed middle of the anchor booth segment). 

Considering claim 48, Ahmed discloses the plurality of elements of the story displayed 
together including the title, description, time and location (see fig 2-b). 

Considering claim 49, Ahmed discloses the name of the summary (see "Erin: A Tropical 
Storm" in fig 2-b) and thus discloses the claimed limitation. 

Considering claim 50, Ahmed discloses the claimed "summary of sentences" (see 
description of "Erin: A Tropical Storm" in fig 2-b). 
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Considering claim 51, Ahmed discloses the claimed summary presentation elements 
comprise a hyperlink to a source media element of the segment (see fig 2-b, hyperlinks 213 and 
214 take user to source of information). 

Considering claim 52, Ahmed discloses the claimed hyperlinks which lead to presentation 
of additional elements of the story segment including named entities (see fig 2-b, selecting 
"Headline News" leads to headline news stories and selecting the sports link 215 leads to sports 
news, stories and named entities). 

Considering claims 53, 54 and 62, Ahmed discloses selecting a hyperlink for searching for 
related stories and presenting a thumbnail view comprising key frames from the multiple story 
segments (see fig 2-b, by selecting "Headline News", news under "Headline News" is searched 
and retrieved and displayed including the "Headline News" keyframe 213 and multiple story 
segments 214). 

Claim Rejections - 35 U.S.C § 103 

3. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness 
rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

4. Claim 14 is rejected under 35 U.S.C. 103(a) as being unpatentable over Ahmed. 
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Considering claim 12, Ahmed discloses cues including a name followed by a comma's in a 
sentence as a means for indicating a break between segments. It would have been obvious 
modifying Ahmed to include ":" would have had the same effect of quickly identifying a break in a 
segment. Therefore, it would have been obvious to one having ordinary skill in the art at the time 
the invention was made to modify Ahmed to include the claimed ":" to quickly identify a break in 
a segment. 

Considering claim 14, Ahmed discloses cues including a token phrase "Jane Doe, WXYZ 
news, reporting live from Anytown, USA" to indicate a break between segments. It would have 
been obvious modifying Ahmed to include "I'm" followed by the person's name would have also 
provided a cue indicating a break between segments since news reporters often identify 
themselves by "I'm" followed by their name. Therefore, it would have been obvious to one 
having ordinary skill in the art at the time the invention was made to modify Ahmed to include the 
claimed "I'm" followed by a name to quickly indicate a break between segments since this phase 
is often used by reporters. 

Considering claim 16, Ahmed discloses cues including a name followed by a comma's in a 
sentence as a means for indicating a break between segments. It would have been obvious 
modifying Ahmed to include a name followed by ":" would have had the same effect of quickly 
identifying a break in a segment. Therefore, it would have been obvious to one having ordinary 
skill in the art at the time the invention was made to modify Ahmed to include the claimed name 
followed by ":" to quickly identify a break in a segment. 



Application/Control Number: 09/033,268 
Art Unit: 2611 



Page 1 1 



Considering claim 1 8, Ahmed discloses cues including a token phrase "Jane Doe, WXYZ 
news, reporting live from Anytown, USA" to indicate a break between segments. It would have 
been obvious modifying Ahmed to include "I'm" would have also provided a cue indicating a 
break between segments since news reporters often identify themselves by "I'm" followed by their 
name. Therefore, it would have been obvious to one having ordinary skill in the art at the time 
the invention was made to modify Ahmed to include the claimed "I'm" to quickly indicate a break 
between segments since this phase is often used by reporters. 

Considering claim 25, Ahmed discloses using token phrases like "we go now to" and 
"more on that". It would have been obvious token phrases like "coming up" and "next on" would 
have also indicated a break between segments since these phrases are often used in the news. 
Therefore, it would have been obvious to one having ordinary skill in the art at the time the 
invention was made to modify Ahmed to include the claimed phrases to quick identify breaks or 
transitions in the segment since these phrases are often used. 

Considering claim 26, Ahmed discloses the claimed sign off phrase (see col 23 lines 55-59 
"Jane Doe, WXYZ news, reporting live from Anytown, USA"). 

Considering claim 27, Ahmed discloses using token phrases like "we go now to" and 
"more on that" to indicate and identify a break or transition in a segment. It would have been 
obvious token phrases like "that is all" or "that's all" would have also indicated a break between 
segments since these phrases are often used in the news. Therefore, it would have been obvious 
to one having ordinary skill in the art at the time the invention was made to modify Ahmed to 
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or transitions in the segment since these 



5. Claims 55 - 57 are rejected under 35 U.S.C. 103(a) as being unpatentable over Ahmed in 
view of Youman et al (5,629,733). 

Regarding claim 55, claim 55 recites some of the same limitations as recited above, and 
thus is rejected for at least those reasons provided above. Claim 55 further recites presenting a 
plurality of related story segments using a layered hierarchical presentation of the summary 
representations at a relatively high hierarchical level and the source data at a relatively low 
hierarchical level, together with hyperlinks permitting navigation among related story segments to 
a desired hierarchical level representation. 

Ahmed discloses displaying a plurality of related summary story segments (see stories for 
"Headline News" in fig 2-b) with source information (see "Headline News" icon 213 on the right) 
and hyperlinks 215 at the top (see fig 2-b). 

Youman teaches displaying hierarchical levels with a summary of a program located at the 
top of the screen with the source information at the bottom (see fig 24a). It would have been 
obvious from Youman placing the source information on a lower level of the screen would have 
provide a better viewing of the summary since the source information would not be in the way i.e. 
in the lower corner. Therefore, it would have been obvious to one having ordinary skill in the art 
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at the time the invention was made to modify Ahmed to include the claimed hierarchical level to 
place the more important information like the summary information on top and the source of the 
information on the bottom to provide a more clearly prominent display of the important summary 
information. 

Considering claim 56, Ahmed discloses the claimed information is a text stream, the 
source data is text relating to a story segment and the summary representation includes named 
entities (see fig 2-b, named entities met by "Erin: A Tropical Storm" the story segment is met by 
the summary description of "Erin: A Tropical Storm"). 

Considering claim 57, Ahmed discloses viewing image data (see fig 2-b) and thus discloses 
the claimed "information stream is an image stream". Further, Ahmed discloses displaying frames 
of information including a title and summary (see item 214 fig 2-b, frame 214 meets "keyframe" 
limitation). 



6. Claims 59, 60 and 61 are objected to as being dependent upon a rejected base claim, but 
would be allowable if rewritten in independent form including all of the limitations of the base 
claim and any intervening claims. 



Allowable Subject Matter 




• 
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Conclusion 

7. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. 

Janse et al (6,340,971) - key-frame based video display 
Blum et al (5,918,223) - segmentation of audio information 
Liou et al (6,278,446) - organization and browsing of video 
Yeo et al (5,821,945) - browsing based on content and structure 
Wissner (5,752,029) - editing multimedia composition 

Any response to this action should be mailed to: 

Commissioner of Patents and Trademarks 
Washington, D.C. 20231 



or faxed to: 



(703) 308-9051, (for formal communications intended for entry) 



Or: 



(703) 308- 5399 (for informal or draft communications, please label 



ii 



PROPOSED" or "DRAFT") 



Hand-delivered responses should be brought to Crystal Park II, 2121 Crystal 



Drive, Arlington. VA., Sixth Floor (Receptionist). 
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Any inquiry concerning this communication or earlier communications from the examiner 
should be directed to Vivek Srivastava whose telephone number is (703) 305 - 4038. The 
examiner can normally be reached on Monday - Thursday from 8:00 am to 5:30 pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Andy Faile, can be reached at (703) 305 - 4380. 

Any inquiry of a general nature or relating to the status of this application or proceeding 
should be directed to the group receptionist whose telephone number is (703) 305 - 3900. 



VS 9/30/02 




